Aiming at the balance optimization problem of Lightweight Convolutional Neural Network （LCNN） in accuracy and complexity， an adaptive multi-scale feature channel grouping optimization algorithm based on fast Non-dominated Sorting Genetic Algorithm （NSGA-Ⅱ） was proposed to optimize the feature channel grouping structure of LCNN. Firstly， the complexity minimization and accuracy maximization of the feature fusion layer structure in LCNN were regarded as two optimization objectives， and the dual-objective function modeling and theoretical analysis were carried out. Then， a LCNN structure optimization framework based on NSGA-Ⅱ was designed， and an adaptive grouping layer based on NSGA-Ⅱ was added to deep convolution layer in original LCNN structure， thus constructing an Adaptive Multi-scale Feature Fusion Network based on NSGA2 （NSGA2-AMFFNetwork）. Experimental results on image classification datasets show that compared with the manually designed network structure M_blockNet_v1， NSGA2-AMFFNetwork has the average accuracy improved by 1.220 2 percentage points， and the running time decreased by 41.07%. This above indicates that the proposed optimization algorithm can balance the complexity and accuracy of LCNN， and also provide more options for network structure with balanced performance for ordinary users who lack domain knowledge.